Setting the threshold for high throughput detectors: A mathematical approach for ensembles of dynamic, heterogeneous, probabilistic anomaly detectors

摘要

Anomaly detection (AD) has garnered ample attention in security research, assuch algorithms complement existing signature-based methods but promisedetection of never-before-seen attacks. Cyber operations manage a high volumeof heterogeneous log data; hence, AD in such operations involves multiple(e.g., per IP, per data type) ensembles of detectors modeling heterogeneouscharacteristics (e.g., rate, size, type) often with adaptive online modelsproducing alerts in near real time. Because of high data volume, setting thethreshold for each detector in such a system is an essential yet underdevelopedconfiguration issue that, if slightly mistuned, can leave the system useless,either producing a myriad of alerts and flooding downstream systems, or givingnone. In this work, we build on the foundations of Ferragut et al. to provide aset of rigorous results for understanding the relationship between thresholdvalues and alert quantities, and we propose an algorithm for setting thethreshold in practice. Specifically, we give an algorithm for setting thethreshold of multiple, heterogeneous, possibly dynamic detectors completely apriori, in principle. Indeed, if the underlying distribution of the incomingdata is known (closely estimated), the algorithm provides provably manageablethresholds. If the distribution is unknown (e.g., has changed over time) ouranalysis reveals how the model distribution differs from the actualdistribution, indicating a period of model refitting is necessary. We provideempirical experiments showing the efficacy of the capability by regulating thealert rate of a system with $\approx$2,500 adaptive detectors scoring over 1.5Mevents in 5 hours. Further, we demonstrate on the real network data anddetection framework of Harshaw et al. the alternative case, showing how theinability to regulate alerts indicates the detection model is a bad fit to thedata.